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ABSTRACT 



This thesis describes the level of User Information 
Satisfaction with the Composite Health Care System (CHCS) . A 
short-form User Information Satisfaction questionnaire, 
developed by Baroudi and Orlikowski (1988) , was administered 
to CHCS users at the Naval Hospital, Charleston. 

Dividing the users into three work groups: Physicians, 
Administrative, and Ancillary personnel; statistically 
significant differences in satisfaction between groups were 
found. Overall, physicians were least satisfied and 
administrative personnel were the most satisfied of the 
groups . 

The respondents as a whole showed noticeable differences 
in satisfaction with factors that comprise User Information 
Satisfaction. The respondents were dissatisfied with the 
factor relating to software contractor's services. However, 
the users were very satisfied with the factor describing local 
Management Information Department services. 
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I. 



INTRODUCTION 



A. OVERVIEW 

Organizations of all types are seeking to improve their 
effectiveness and efficiency by using information systems. 
Considerable amounts of time and money are spent to develop 
and implement information systems within an organization. 
Once inplemented, an information system may or may not be 
effective in accomplishing its objective. Thus, an important 
and necessary stage in justifying any particular information 
system is to determine its value of effectiveness to the 
organization . 

The Department of Defense (DoD) has undertaken the 
development and implementation of a large information system 
for use at Medical Treatment Facilities. The system is called 
the Composite Health Care System (CHCS) . Due to the costs 
associated with this system, a simple and efficient measure is 
needed to analyze the effectiveness of this information 
system. 

Selecting the appropriate measure of system effectiveness 
is difficult. Much emphasis is placed in many organizations 
on economic analysis of information system. This type of 
analysis provides only an economic measure of effectiveness 
for a system: will the benefit outweigh the cost? If the 
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objective of the organization is not purely economic, the 
measure is not complete. 

Many researchers have proposed measuring the level of 
satisfaction perceived by the users of the system as a 
surrogate measure of effectiveness. No matter how technically 
correct an information system, if the users are not satisfied 
with the system, it is not an effective system. Conversely, 
if the users are satisfied, the system is considered effective 
in meeting their needs. 

B . RESEARCH OBJECTIVES 

This study will investigate through empirical research 
whether the Composite Health Care System is effective through 
the use of the user information satisfaction surrogate 
measure. Using a previously developed, documented, and 
validated survey instrument, this study will describe the user 
information satisfaction characteristics at one implementation 
site of the Composite Health Care System: Naval Hospital, 
Charleston. The results of this survey will help pinpoint 
potential problem areas with the system implementation as well 
as document areas without problems. Once completed, the 
results will serve as a baseline gauge to compare satisfaction 
with CHCS at other facilities and to measure changes in 
satisfaction at Charleston at a later time. 
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II. DESCRIPTION OF THE COMPOSITE HEALTH CARE SYSTEM 



A. BACKGROUND 

The Department of Defense (DoD) has pursued the goal of 
providing automated computer support to its hospitals and 
clinics since 1968. In February 1979, the Mission Elements 
Need Statement (MENS) , establishing the need for an automated 
computer system was approved. Under the direction of the Tri- 
Service Medical Information System (TRIMIS) program office, 
stand-alone and integrated health care computer systems were 
acquired, implemented and operated to support Pharmacy 
(TRIPHARM) , Laboratory (TRILAB) , Radiology (TRIRAD) , 
Appointment and Scheduling (TRIPASS) , Quality Assurance 
(AQCESS) , and Hospital Information System (HIS) . The 
knowledge and experience gained through the operation of these 
systems since 1979 was used to refine and validate the system 
requirements for a completely new and fully integrated 
information system: The Composite Health Care System (CHCS) . 

B . CHCS PURPOSE 

The primary purpose of CHCS is "to provide health care 
services that support military forces in fulfilling their 
required mission...." (Draft System Decision Paper, 1989). 
Improving the efficiency and effectiveness of military health 
care delivery during peacetime and during mobilization through 
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integration of information resources is the goal of CHCS . The 
major focus of CHCS is the integration of shared information 
resources to resolve the deficiencies of communication, 
decision support, and information processing. The Composite 
Health Care System is designed as a fully integrated medical 
information system that provides automated support of 
information requirements for military medical treatment 
facilities. CHCS supports the administrative functions and 
the delivery of health care with information retrieval 
services . 

C . DEPLOYMENT STATUS 

CHCS is undergoing the Operational Test and Evaluation 
(OT&E) phase and is currently deployed to Naval Hospital 
Charleston as one beta-test site. The role of a beta-test 
site is to incrementally replace the TRIMIS systems with 
validated CHCS software. Continued validation of requirements 
and extensive evaluation of the CHCS software is conducted 
prior to the decision to deploy CHCS worldwide. 

D . OPERATIONAL FEATURES 

CHCS supports various functional areas in sharing 
information. Each functional area uses this information for 
its own purpose and communicates results/activities to other 
areas. Functional areas supported by CHCS include: 
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1 . Patient Administration 



Patient administration does the registration of a 
patient into the system. Once in the computer system, the 
patient's demographic information is available for access by 
all other modules. The patient administration module also 
performs the functions to admit patients to the hospital, 
transfer patients between wards, and discharge patients from 
the hospital. The creation, update and closing of inpatient 
records is also performed by the patient administration 
module . 

2 . Patient Appointment and Scheduling 

The Patient Appointment and Scheduling module provides 
for a centralized appointment service, a decentralized 
appointment service, or a combination of both depending on the 
desire of the hospital. The appointment service creates and 
maintains the appointment schedule for the hospital. A 
patient is given an appointment date and time to see the 
health care provider creating a centralized repository of 
information to produce operational reports of activity. 

3 . Nursing 

CHCS supports inpatient nursing in a variety of areas. 
In the area of patient care, nursing uses the system for 
entering and obtaining the status of all physician orders, 
entering patient assessment data, and generating patient care 
plans. Documentation of patient progress is performed in the 



5 



automated nursing notes. Nursing unit management uses CHCS to 
generate shift care plans, patient management reports, drug 
administration time reports, and staffing requirement reports. 
Routine administrative functions are performed by CHCS for 
nursing including: staff credentialling, documentation of 
continuing education and in-service training, and staff 
scheduling capabilities. 

4 . Laboratory 

The laboratory uses the CHCS system for processing of 
orders for laboratory tests with automated reporting of test 
results. Specimens are processed, tracked, and reported 
through the system. All results are placed into the patients 
automated medical record and are immediately available for 
inquiry by health care providers. Automated control of blood 
bank operations including: blood acquisition, inventory and 
utilization are accomplished through CHCS. The laboratory is 
supported with ability to manage inventory control, register 
patients into the tumor registry, conduct a drug testing 
program, and generate a multitude of management reports. 

5 . Pharmacy 

The pharmacy uses CHCS to process prescriptions for 
patients that are ordered by health care providers. The 
health care provider enters the prescription information at a 
terminal located in his office. The information is 
electronically transmitted to the pharmacy. The pharmacy can 
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immediately prepare the medication for dispensing before the 
patient leaves the physician's office. CHCS automatically 
performs checks for patient allergies, drug interactions with 
medications the patient is currently taking, drug dosage for 
the age of the patient, and records the prescription into the 
patient's medical record. In-patient order processing is 
accomplished in much the same fashion. On-line drug 
monographs are available for inquiry by medical staff at any 
terminal. Pharmacy management of inventory control and 
reporting are also accomplished. 

6 . Radiology 

Orders for radiological procedures are electronically 
transmitted to the radiology department where the order is 
processed. Order tracking and results reporting are performed 
by the system. Radiology results are immediately available 
for inquiry by medical staff. Department management including 
inventory control is also conducted through CHCS. 

7 . Clinical Dietetics 

The Clinical Dietetics service uses CHCS to obtain 
patient data needed to access the nutritional needs of the 
patient. This information is used to create patient diet 
plans, select patients of interest, patient menu selections 
and monitor patient nutritional data. CHCS is also used to 
maintain the nutrition clinic schedule and compute clinical 
dietetics workload data. 
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E. POTENTIAL BENEFITS 



In general terms, the overall benefits of the CHCS system 
include : 

1. reliable, timely, easily accessible clinical 
information 

2. improved documentation for medicolegal issues 

3. improved communication 

4. improved patient satisfaction 

5. improved management efficiency 
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III. USER INFORMATION SATISFACTION AND ITS MEASUREMENT 



A. INTRODUCTION 

Organizations spend considerable amounts of time and money 
to develop and implement information systems. However, since 
financial resources are scarce and limited, not all 
application systems can be developed. Only those applications 
that can be justified are approved and developed. An 
important and necessary stage in justifying any particular 
information system is to determine its value or effectiveness 
to the organization. 

B. METHODS TO ASSESS EFFECTIVENESS 

Several methods have been suggested in the empirical and 
non-empirical literature for assessing the effectiveness of an 
information system. 

System usage (Swanson, 1974; Conrath and Mignen, 1990) 
assumes that a casual relationship exists between 
effectiveness of an information system and the amount of time 
the system is used. An effective information system is used 
more than a less effective system. This method also assumes 
that the use of the system is optional to the user. If, 
however, the use of the system is mandatory, the relationship 
between use and effectiveness fails. 
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Others (eg., Hamilton and Chervany, 1981) argue that 
effectiveness is determined by comparing performance to 
ojbectives. To assess the effectiveness of an information 
system, the task objective of the system is first determined. 
Criterion measures are developed to measure the extent to 
which the information system accomplishes the determined 
objectives. Problems are encountered with this method 
because: objectives and measures are often not defined 
adequately; efficiency-oriented and easily employed measures 
are frequently used over effectiveness-oriented and difficult 
measures; individual interpretations of what the ojectives and 
measures are, often conflicting, exist. 

Another method frequently used to determine the value of 
an information system is by economic (cost-benefit) 
evaluation. The benefits of a system, by improved 
organizational effectiveness, are weighed against the cost for 
development and operation of the system (Nolan, 1974) . 
Although this appears objective and comprehensive on paper, in 
practice, cost-benefit analysis is highly subjective and 
difficult to conduct (Nolan, 1974) . The difficulties in using 
this approach for research argued by Ives et al. (1983) stem 
from the fact that: (1) many costs and benefits are intangible 
and not easily recognizable and/or converted into monetary 
equivalents, (2) unstructured, ad hoc decision making benefits 
are nearly impossible to objectively assess, (3) even when 
these items are determined by an organization, the data are 
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generally unrecorded and not available for research (Ives et 
al., 1983). 

Theoretically, evaluation of the effectiveness of an 
information system is best accomplished by decision analysis, 
based on its degree of use in decision making and the 
resultant productivity benefits attributable to its use 
(Nolan, 1974) . An effective information system supports a 
user by retrieving necessary information in the proper format, 
level of detail, and at the proper frequency for use. 
However, the decision analysis approach suffers a significant 
drawback for use in scientific research: it is non- 
quantifiable and cannot be easily replicated. 

C. THE USER INFORMATION SATISFACTION CONSTRUCT 

A surrogate for utility in decision making is the 
satisfaction of users of the information system (Neumann and 
Segev, 1980; Ives et al . , 1983; Bailey and Peason, 1983; 
Baroudi and Orlikowski, 1988) . 

Webster' s New Collegiate Dictionary defines satisfaction 
as "the fulfillment of a need or want". Organizational 
Psychologist Bernard Bass (1965) generically defines 
satisfaction to mean "... the extent to which the item is 
rewarding to us relative to how much better we might do 
elsewhere and what aspirations we have." 

The concept of User Information Satisfaction (UIS) can be 
traced back to the work of Cyert and March (1963) . Their 
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research argues that organizational behavior continually 
imposes upon the manager the need for information. If a 
formal information system exits, the success or failure of 
that information system to meet the needs of the user either 
reinforces or frustrates the user's sense of satisfaction with 
that system. 

Figure 3.1, an adaptation of the Cyert and March model, 
portrays the reinforcement /reduction of satisfaction process 
in decision making using the information system. When a user 
perceives that the needed information should be readily 
available in the formal information system, he will use the 
information system in the normal manner to retrieve the 
required information. If the information is readily 

available, satisfaction with the system will be reinforced. 
But, if the information is not readily available, he must 
adapt and conduct a search beyond the information system by ad 
hoc methods or by some other less familiar method. 

An "expanded search" through the information system is a 
cause of frustration to the user because of the time and 
effort used to obtain the necessary information. 
Additionally, the user is frustrated because he perceived the 
information to be readily available when it was not. From 
users' standpoint, information systems may actually block the 
user from carrying out the act of decision making (Nolan, 
1965) . 
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UIS reinforcement/reduction 
in decision making 



The information system is continually being evaluated by 
the user from repetitive use. If over a period of time of 
use, the user perceives that the information system cannot 
retrieve the information without the frequent use of expanded 
searches, frustration will occur. Consequently, this may lead 
to an expression of user dissatisfaction with the information 
system. The project is a failure if the end product does not 
satisfy the manager whom it is to serve (Powers and Dickson, 
1973) . If, however, the user is able to reliably obtain the 
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needed information with normal operation of the information 
system, satisfaction with the system will be reinforced and 
the information system is successful. 

Of note are two empirical studies, (Aldag and Power, 1986/ 
Gallupe and Desanctis, 1988) , undertaken to find a 
relationship between satisfaction and actual system 
performance using objective third party judges. The results 
of these studies, however, are conflicting but do not support 
the assumption that increased satisfaction indicates actual 
system performance increases. 

D. THE MEASUREMENT OF USER INFORMATION SATISFACTION 

Potentially measurable and generally acceptable, several 
studies have employed user information satisfaction (UIS) as 
a dependent variable to indicate system effectiveness and 
acceptance (Ives and Olson, 1979/ Igbaria and Nachman, 1990) . 
Neumann and Segev (1980) show a correlation between user's 
reaction to satisfaction factors and their perception of the 
organizations' s performance . Swanson (1974) empirically found 
high correlation between the user's appreciation of the system 
and his/her usage of the outputs. Powers and Dickson (1973) 
concluded that user satisfaction is the most critical factor 
for system success. Although there is no standard measure of 
satisfaction in these studies, it is argued that user 
information satisfaction is an indicator of system usage and 
success . 
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The level of user information satisfaction in the studies 



is derived from a myriad of factors when users were asked to 
evaluate their computer services relative to a sense of 
satisfaction (Bailey and Pearson, 1983) . Factors measured in 
the various studies include: accuracy, content, frequency, 
timeliness, reliability, assistance, adequacy, accommodation, 
communication, access, appreciation and flexibility. Each of 
these studies use a measure that is unique to that particular 
study (Ives et al., 1983), but taken as a whole provide 
insight into a description of UIS. 

Validation of the UIS measures in these studies is 
limited, especially in handling threats to internal validity 
(Haga and Zviran, 1990) . However, a tool for measuring and 
analyzing UIS was developed by Bailey and Pearson (1983) , 
based on the previous studies, that provides the basis of an 
instrument, later refined (Ives et al . , 1983), and validated 
(Baroudi and Orlikowski, 1988) for use. Although the method 
employed to measure user satisfaction by this tool is a 
pre-experimental design lacking pretest-postest or control 
group (Campbell and Stanley, 1966) , Conrath and Mignen (1990) 
claim it represents a key contribution in the development of 
a standard instrument to measure user satisfaction. The 
development and refinement of this instrument will be 
described in more detail. 

Bailey and Pearson (1983) described a model proposed by 
Lawler and Wanous (1972) for measuring user satisfaction: 
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i 

$i= E R iFi 3 

j=i 

where 

R ±j = The reaction to factor j by the individual i 

W i;) = The importance of factor j to individual i 

Using this model, the satisfaction of the user is measured 
as the weighted sum of the user' s positive and negative 
reactions to a set of factors about the information system 
(Bailey and Peason, 1983) . The user's perception of a "good" 
information system would be a system that the user is very 
satisfied with the factors considered most important in the 
system. 

To implement the model, the set of factors representing 
the domain of user satisfaction was established via a review 
of 22 studies dealing with computer/user interface (Bailey and 
Pearson, 1983) . The initial 36 factors generated from the 
literature review were expanded to 39 factors after further 
review by middle managers. They concluded that the 39 factors 
included represent the domain of user satisfaction at (X=0.01. 

Bailey and Pearson (1983) used four bipolar adjective 
pairs on a seven point Likert-type scale to measure the user' s 
perception of a factor. Along with the four adjective pairs 
was included the user' s perception of importance of the 
factor. The seven intervals from negative to positive were 
denoted by adverbial qualifiers. Figure 3.2 represents their 
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use of the semantic differential technique for measuring the 
user's response to "Format of Output" factor. 

Format of Output : The material design of the layout and 

display of the output contents. 

good : : : : : : : : bad 

simple : : : : : : : : complex 

readable : : : : : : : : unreadable 

useful : : : : : : : : useless 

To me, this factor is 

important : : : : : : : : unimportant 

Figure 3.2 Illustration of questionnaire form 

The Bailey and Pearson user information satisfaction 
questionnaire was determined to be a reliable, valid 
instrument (Bailey and Pearson, 1983) . It represented an 
important first step toward the development of a valid and 
useful UIS measure (Ives et al . , 1983). 

Deese (1979) used the Pearson questionnaire at the Federal 
Computer Performance Evaluation and Simulation Center. He 
stated, "The results identified problems that would not 
otherwise have been discovered" and claimed that the user 
satisfaction questionnaire was a very useful and worthwhile 
tool (Deese, 1979) . 

Ives, Olson and Baroudi (1983) chose to undertake an 
in-depth assessment of the Pearson (1983) questionnaire. 
Replicating the initial Bailey and Pearson study, they were 
able to reinforce the validity and reliability of the 
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instrument. They presented several approaches to improve the 
quality of the original Pearson instrument. 

The goal of the suggested improvements was to establish a 
standardized "short form" instrument. Since Pearson found the 
importance scale provided no additional information, it was 
eliminated from the instrument. Scales which showed 
undesirable psychometric qualities were also eliminated. The 
number of items scaled within a question was reduced from four 
to two in order to reduce the time to complete the 
questionnaire. All of the scales in the original Pearson 
instrument were scored positively to the left and negatively 
to the right end of the scale. To reduce the undesired 
effects created by a person simply marking down a column of 
responses, some of the scales were reversed scored, thereby 
increasing the reliability of the measure. The resulting 
short form was determined to substantially measure the concept 
of UIS originally proposed in the Pearson full instrument 
(Ives et al., 1983). 

The short form measure (Ives et al., 1983) was further 
subjected to testing by Baroudi and Orlikowski (1988) to 
examine its psychometric properties. The final short form 
questionnaire consisted of 13 questions with two items 
evaluated per question (Appendix A) . Each item is scored on 
a seven point Likert-type scale from -3 (dissatisfied) to +3 
(satisfied) with zero indicating a neutral response. 



18 



The total individual user information satisfaction score 



is calculated by averaging the responses of the two items for 
each question and summing the scores of the 13 questions. 




Ki + J ? 2 

2 



where 

S ± = total individual user satisfaction 
R-! = response to the first item scale 
R 2 = response to the second item scale 

The range of total satisfaction can be from -39 to +39. 

Three factors/ electronic data processing (EDP) staff and 
services (Factor A) , information product (Factor B) , and 
knowledge and involvement (Factor C) ; were found to comprise 
user satisfaction using factor analysis by Ives, Olson and 
Baroudi (1983) . These subtotals are calculated as the average 
of the responses to questions loading into a particular 
factor. Questions 1, 2, 6, 11, and 12 load heavily into the 

electronic data processing (EDP) staff and services factor. 
Questions 7, 8, 9, 10 and 13 load heavily into the information 
product factor. Questions 2, 3, 4, and 5 load heavily into 

the knowledge and involvement factor. The factor subtotals 
will range from -3 to +3 in value. Averages are used for 
meaningful comparison between the three factor scores. 

Data were gathered for the study from 358 employees, 
mostly clerical and support personnel, of 26 New York area 
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organizations. Construct validity was determined first by 
examining the relationship between each scale and the total 
UIS score. Factor analysis using varimax rotation converged 
to a three factor solution accounting for 68% of the variation 
in five iterations. This provides strong evidence of 
construct validity (Baroudi and Orlikowski, 1988) . 

Convergent validity was proven by comparing an interview 
measure of satisfaction with the instrument measure in two 
groups. One group of users were generally pleased and 
satisfied with the information system and the other group of 
users were generally dissatisfied on interview. 
Administration of the instrument indicated a statistically 
significant difference by t-test in the groups at p < .001 
(Baroudi and Orlikowski, 1988) . 

The total satisfaction and subtotal scores reported a 
reliability level above the .80 required for research. This 
demonstrates the instrument is internally consistent and 
reasonably free from measurement error. 

The work by Baroudi and Orlikowski (1988) and resulting 
short- form UIS instrument provides a reliable and valid 
measure of user information satisfaction. It furnishes a 
meaningful, standardized measure of the overall satisfaction 
with an information system as well as specific information 
about satisfaction within the subfactors of electronic data 
processing (EDP ) staff and services, information product, and 
user knowledge and involvement. 
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IV. 



RESEARCH METHODOLOGY 



A. INTRODUCTION 

The purpose of this research is to identify the 
characteristics of user information satisfaction (UIS) for 
users of the Composite Health Care System (CHCS) . The study 
is based on statistical analysis of empirical data collected 
for CHCS users at the Naval Hospital, Charleston, SC. The 
following sections will discuss the survey instrument, data 
collection methodology and methods of statistical analysis 
employed in the study. 

B. THE SURVEY INSTRUMENT 

To avoid the pitfalls associated with developing a 
comprehensive survey questionnaire, the previously developed, 
psychometrically evaluated and validated short-form 
questionnaire of Baroudi and Orlikowski (1988) was used 
without alteration. A copy of the complete survey 
questionnaire is included in Appendix A. The questionnaire is 
composed of two sections: a demographical section and a user 
information satisfaction section. 

1 . Demographic Section 

The first section of the questionnaire requested 
general information. Questions 1-5, the respondents were 
asked to give their hospital department, job description, 
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highest level of education, age, and gender respectively. To 
determine the user' s experience level with the system, the 
sixth question asked for the length of time, in months, the 
respondent had used CHCS . Further determination of the user's 
experience was sought in questions seven and eight. Question 
seven asked if the user had used other computer systems 
previously. If the respondent answered "yes" to question 
seven, he or she was asked if the previous system was a health 
care information system. 

2 . User Information Satisfaction Section 

The second part of the survey directly addressed the 
issues of user information satisfaction. The instrument was 
designed to elicit the user's perception of how well the CHCS 
system functioned in their area. It consisted of thirteen 
questions; each question having two bipolar adjective Likert- 
type item scales for responses. Each item can take on one of 
seven values from -3 (extremely dissatisfied) to +3 (extremely 
satisfied) with zero indicating a neutral response. 

The questions asked can be categorized into one of 
three factors of user satisfaction. 

Management Information Services (MIS) Staff and 
Services ; This factor is the respondent's perception of the 
attitude and responsiveness of the MIS staff and the 
relationship with the MIS staff. Due to the fact that the MIS 
personnel at the Naval Hospital, Charleston provide only local 
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support and act as liaison to the contractor that supplies the 
software product, this factor is further broken down to 
reflect services provided by the local MIS department and the 
contractor . 

Information Product: This factor is the respondent's 
perception of the quality of output delivered by CHCS. 

Knowledge and Involvement: This factor is the 
respondent's perception of the quality of training provided, 
their understanding of the system, and their participation in 
its development. 

C. SAMPLE AND DATA COLLECTION 

Naval Hospital, Charleston is one of thirteen beta-test 
sites for CHCS software. It was selected as the site to 
conduct the survey because all functional modules were to be 
operational at the time of the survey. Since the in-patient 
modules, servicing nursing and clinical dietetics, were not 
implemented at the time of data collection, the sample 
population then consisted of personnel from out-patient and 
administration areas. 

A local point of contact acted as a distribution and 
collection agent for the surveys. The questionnaires, each 
accompanied with a cover letter (Appendix B) , were mailed to 
the point of contact at the site. The point of contact 
distributed 180 questionnaires and return envelopes to all 
department personnel within the hospital operating the CHCS 
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modules. The surveys were completed and returned by the 
respondents to the point of contact in sealed envelopes and 
then the questionnaires were forwarded to the researcher for 



analysis . 

110 questionnaires were returned, of which 101 surveys had 
complete information. This represents a final return rate of 
56%. 

D. STATISTICAL ANALYSIS METHODS 

Data from completed surveys were coded and entered into a 
spreadsheet program. The spreadsheet program was used to 
perform the statistical analysis. The specific procedures 
used in the analysis will be presented next. 

1. Respondent's Satisfaction Index 



Each respondent's total satisfaction index was 



calculated for the 13 questions using the following formula: 



where : 

Si = Satisfaction Index for respondent i 
R aij = Response to first item scale of question j for 
repondent i 

Rbij = Response to second item scale of question j for 
respondent i 
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2 . Total Satisfaction Index 

The overall or total satisfaction index for the survey 
group is calculated by averaging the respondent's satisfaction 
index to find the mean. 

3 . Question Averages 

The mean response to each of the thirteen questions is 
calculated for comparison of individual questions. 

4 . Factor Subtotals 

The thirteen questions can be grouped into three 
factor subtotals as was defined by Ives, Olson, and Baroudi 
(1983) : MIS staff and services (Factor A) ; Information product 
(Factor B) ; and Knowledge and involvement (Factor C) . The 
mean of the individual questions averages is calculated to 
find the factor subtotals. Questions 1, 2, 6, 11, and 12 are 
used for the MIS staff and services subtotal. Questions 7, 8, 
9, 10, and 13 are used for the information product subtotal. 
Questions 2, 3, 4, and 5 are used for the knowledge and 

involvement subtotal. 

The MIS staff and services sub-factor is further 
subdivided to provide insight into the distinction between the 
local Management Information Department (MID) department 
functions and the functions of the software contractor. 
Again, the mean of the individual question averages is 
calculated to subdivide the MIS staff and services factor. 
Questions 1, 6, and 11 directly address the attitude, services 
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and relationship with the local MID are used for the local MID 
factor. Questions 2 and 12 address issues associated with 
software developement , the responsibility of the civilian 
contractor, and are used for the contractor factor. 

5 . Comparison of Groups 

The respondents were categorized by the type of use of 
the CHCS system. Three work groups were identified: (1) 
Physicians; (2) Ancillary; and (3) Administration. The 
individual question averages, total satisfaction indexes, and 
sub- factor totals were compared for one group verses the 
combination of the other two groups using a t-test with the 
significance level of alpha = 0.05. The t-test is used to 
determine if there is a significant difference between the 
arithmetic mean value of two groups. 
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V. DESCRIPTIVE FINDINGS 



A. DEMOGRAPHIC FINDINGS 

Of the 101 respondents, 57 were male and 44 were female. 
The age of the respondents ranged from 19 to 56 with a mean of 
32 years old. 

1 . Work Center Distribution 

The hospital department work centers reported were 
out-patient areas including: administration, clinics, pharmacy 
and laboratory departments. The CHCS modules for in-patient 
areas were not operational at the time of the survey. 
Personnel from in-patient areas were not included in the 
study. Figure 5.1 reflects the distribution by work center. 



DEPARTMENT WORK CENTERS 




Figure 5 . 1 
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2. Job Description Distribution 



The job descriptions reported are from a multitude of 
specific functions but can be categorized as: 

1. Technician - a hospital corpsman functioning in a 
medical-technical capacity (pharmacy technician, 
laboratory technician, clinic corpsman) 

2 . Physician - a medical doctor 

3. Health Professional - a licensed medical 
professional other than a physician (nurse, 
pharmacist, physical therapist, bio-medical 
officer) 

4. Other - a person not listed above (hospital 
administration non-medical clinic staff) 

Figure 5.2 reflects the distribution by job description. 



JOB DESCRIPTION 

TECHNICIAN 




Figure 5 . 2 
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3 . Work Group Distribution 

For the purposes of this investigation, the study 
population was segregated according to the assumed primary use 
of the computer system. The categories provide a clear 
separation of work groups from an organizational and cultural 
standpoint. These groups are used extensively throughout the 
study. 

1. Administrative - use of the system for indirect 
medical purposes (patient registration, 
appointment scheduling and administrative 
reporting) 

2. Ancillary - use of the system for direct medical 
support (prescription filling, laboratory specimen 
processing) 

3. Physician - use of the system by physicians 
(accessing patient data, transmission of orders) 

Figure 5.3 represents the population of respondents according 
to their main use of the system. 
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WORK GROUP 

by primary system use 




Figure 5.3 



4 . Education Distribution 

The level of education reported by the respondents 
represents a well educated user population. Almost half of 
the users have a college degree and almost ninety percent have 
had some college education. Figure 5.4 represents the 
breakdown by education. 



> 
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EDUCATION 




MASTER DEG. 
6 



Figure 5 . 4 



B. COMPUTER SYSTEM USE 

The length of time the respondents reported as having used 
the CHCS computer system ranged from one month to 18 months 
with a mean of 8.7 months. Of the 101 respondents, 83 had 
used a computer system previously. Only 24 of the 83 users 
had used a health care information system before the CHCS 
implementation. Although many of the respondents had 
experience with computers, this was the first exposure to a 
health care information system for 76 percent of the users. 

C. SATISFACTION FINDINGS 

The survey results for user satisfaction are compared for 
the three work groups in three areas: 1) overall satisfaction, 
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2) responses to individual questions, 3) three group factors: 
MIS staff and services, information product, and knowledge and 
involvement. The actual values reported in the survey 

responses are not significant in themselves, rather the 
relative scores among individual questions, sub-factors and 
work groups are useful for comparison. Since ancillary 
composes 46% of the survey group, the survey averages will 
tend to more closely reflect the responses of the ancillary 
group than the other groups. Therefore, average scores on 
individual questions are not fully representative and are 
included only for rough comparisons. 

1 . Overall Satisfaction 

Overall satisfaction is measured by the mean of the 
sum of the responses to the survey questions. The mean survey 
score index was 11.26 on a scale ranging from -39 to +39. 
Figure 5.5 presents a comparison of the overall satisfaction 
index the three groups and the overall mean satisfaction. The 
survey shows that the administrative group displays the 
highest overall satisfaction (14.96), followed by ancillary 
(9.65), with physicians recording the least overall 
satisfaction (8.07) . Using a t-test, there is no significant 
difference between any one work group and the remaining groups 
at alpha = 0.05. 
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OVERALL SATISFACTION 
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Figure 5.5 



2 . Individual Questions 

Each of the 13 questions on the survey has two 
responses from the user. The response scales were reversed 
scored on many of the questions to neutralize the effect of an 
individual marking boxes straight down the questionnaire. 
Figure 5.6 shows the average level of satisfaction for 
individual questions by work group. 
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Individual Question Response 

By Work Group 
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Figure 5 . 6 



It is readily apparent that some questions exhibit a 
relatively high or low level of satisfaction when compared to 
the other questions. Three questions (question 1, 6 and 11) 
stand out from the others for the high level of satisfaction 
expressed by the users. Each of these questions deal with 
issues concerning the Management Information Services (MIS) 
staff. Two questions stand out as exhibiting an extremely low 
(negative) satisfaction index. Questions 2 and 12 are 
concerned with software development time. A complete 
decomposition of each question by work group will follow. 

Figure 5.7 displays the responses to individual 
questions by physicians compared to the remainder of the 
respondents; Figure 5.8 administrative compared to the 
remainder; Figure 5.9 ancillary compared to the remainder. 
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individual Question Response 

Physician vs Others 
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Figure 5.7 



Individual Question Response 

Administration vs Others 
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Figure 5 . 8 
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a. Question 1 - Relationship with the Management 

Information Department (MID) Staff 

This question measures the level of constructive 
interaction between the user and the Management Information 
Department. As noted earlier, the level of satisfaction 
towards the MID staff is relatively high. Although not 
statistically significant at alpha = 0.05, the administrative 
group showed the highest level of satisfaction of the three 
groups. Slightly lower was the ancillary group followed by 
the physicians. 

b. Question 2 - Processing of Requests for Changes to 
Existing Systems 

The requested responses were concerned with the 
length of time required and timeliness of changes to the CHCS 
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system. The overwhelmingly negative level of satisfaction; 
the lowest scoring question; indicates a perceived problem by 
the users. Ancillary users were the least satisfied, 
physicians slightly higher and administrative users, albeit 
still negative, the highest group. A t-test indicated no 
significant differences at alpha = 0.05. 

c. Question 3 - Degree of Training Provided to Users 
This question asked if the users were completely 

trained to use the system. The level of satisfaction of the 
administrative group was statistically higher at alpha = 0.05 
than the other groups combined. Correspondingly, the 
ancillary group was significantly less satisfied with the 
training. The physicians had the lowest mean score but did 
not prove to be statistically different. 

d. Question 4 - User's Understanding of the System 
The level at which a users understands a system 

naturally flows from training and experience. As expected, 
the response to this question closely parallels the users 
perception of the degree of training provided. In this case, 
physicians were significantly less satisfied at alpha = 0.05. 
Of particular note is the significant improvement in 
satisfaction in the ancillary group from their satisfaction of 
training. 
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e. Question 5 - User's Feeling of Participation 

The user' s perception of participation is 
beneficial for the success of a computer system. Lack of 
participation may lead to dissatisfaction with the system. In 
response to this question, the administrative group had a 
statistically significant higher feeling of participation than 
the other groups. Physicians perceived the least amount of 
participation. Ancillary personnel responded more positively 
than physicians but less positively than administrative. 

f . Question 6 - Attitude of the Management Information 
Department (MID) Staff 

The willingness and commitment of MID staff to lend 
assistance for the benefit of the users is measured by their 
perceived attitude towards the users. The overall response to 
this question (the highest rated question) indicates that MID 
displays a highly cooperative and positive attitude. All 
three groups responded favorably. However, administration 
reported a statistically significant (alpha = 0.05) higher 
response than the other groups. Although the physicians 
scored this question the highest of all 13 questions, their 
response was statistically lower than the other groups. 

g. Question 7 - Reliability of Output Information 
Users must be able to accept the output information 

provided by the computer system as reliable. This is 
extremely important for medical information systems. The 
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responses were positive with no significant differences among 
the groups. Of note is the fact that physicians responded 
more favorably than the other groups and ancillary the least 
favorably . 

h. Question 8 - Relevancy of Output Information (to 
intended function) 

This question measures the degree of congruence 
between what the user wants or requires and what is provided 
by the information products and services. All groups 
responded positively without statistically significant 
differences. The administrative group was most satisfied with 
the relevancy of output of the three groups. 

i . Question 9 - Accuracy of Output Information 

All three groups were satisfied that the system was 
providing correct information. Of the three groups, the 
physicians were the most satisfied with the accuracy of output 
information. Ancillary and administrative groups were about 
equally satisfied. The differences in responses were not 
statistically significant. 

j. Question 10 - Precision of Output Information 
Precision of output information is the variability 

of output information from that which it purports to measure. 
The response was favorable, but was the lowest of the 
questions answered positively. The ancillary group responded 
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statistically lower than the other groups on this question 
while the physicians recorded the highest score. 

k. Question 11 - Communication with the Management 

Information Department (MID) Staff 

Communication is the manner and methods of 
information exchange between the users and the MID staff. As 
noted previously, all questions concerning MID staff received 
high satisfaction levels from all the groups. None of the 
groups' responses were statistically different by t-test at 
alpha = 0.05. 

l. Question 12 — Time Required for New Systems 

Developement 

New systems development time is the time between 
user-initiated request for service or action and a reply to 
that request. Much like the processing of requests for 
changes to existing systems, the groups were either neutral or 
highly dissatisfied. The physicians were statistically less 
satisfied than the other groups/ reporting the lowest index of 
any question. 

m. Question 13 - Completeness of Output 

All three groups were adequately satisfied with the 
comprehensiveness of the content of information product. The 
level of satisfaction of each group was similar with the 
administrative group the most satisfied. 
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3 . Grouped Factors 



The questions are grouped into three factors that 
comprise user satisfaction described by Ives, Olsen, and 
Baroudi (1983) . These factors: MID staff and services, 
information product, and knowledge and involvement, are 
calculated through averages of the component questions. MID 
staff and services (Factor A) is sub-divided into two separate 
components: local MID staff functions (MID) and contractor 
functions (Contractor) . Factor A (MID) is derived from 
questions 1, 6, and 11; Factor A (Contractor) from questions 2 
and 12; information product (Factor B) from questions 7, 8, 9, 
10 and 13; and knowledge and involvement (Factor C) from 
questions 2, 3, 4, and 5. 

None of the work groups' satisfaction level to any of 
the factors were statistically different at significance level 
alpha = 0.05. However, general observations can be made. 
Figure 5.10 represents the work groups' level of satisfaction 
with relation to sub-total factors. 
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Grouped Factors 

By Work Group 
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Figure 5.10 



Figure 5.11 represents factor satisfaction of 
physicians compared to other respondents. 
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Figure 5.12 represents factor satisfaction of 
administrative personnel compared to other respondents. 



Grouped Factors 

Administrative vs Others 
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Figure 5 . 12 



Figure 5.13 represents factor satisfaction of 
ancillary personnel compared to other respondents . 
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Grouped Factors 
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a. Factor A (Local MID Staff and Services) 

This factor represents the attitude, services and 
relationship with the local MID personnel. As seen in Figure 
5.10, the administrative personnel displayed the highest level 
of satisfaction. The lowest level of satisfaction, although 
still extremely positive, was displayed by the physicians, 
jb. Factor A (Contractor Services) 

This factor represents the responsiveness and 
services of software programmers. This factor received the 
lowest level of satisfaction of any factor. All work groups 
rated this factor negatively, with ancillary personnel 
recording the lowest level of satisfaction of any group. 
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c. Factor B - Information Product 



This factor represents the users perception of the 
quality of output from the system. Although the differences 
among the groups were not statistically significant, the 
administrative group scored the highest level of satisfaction. 
Ancillary members presented the lowest level of satisfaction. 

d. Factor C - Knowledge and Involvement 

This factor represents the respondents' self- 
reported assessment of the quality of training provided, 
their understanding of the system, and their participation in 
its development. There was a statistically significant 
difference between the level of satisfaction of the 
administrative group and the other respondents. 
Administrative personnel were highly satisfied with their 
knowledge and involvement of the system. 

4 . Time of System Use 

As users become familiar with the functions of a 
system over time, it is expected that the more experienced 
users would exhibit a greater level of satisfaction. 

Separating the respondents into subgroups of users 
with experience less than six months, six months to 11 months, 
and greater than 11 months provides subgroups of approximately 
equal size. The work groups exhibit roughly the same 
proportions. Figure 5.14 represents the breakdown of 
respondents time of system use. 
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Time of System Use 




Figure 5 . 14 



All the respondents, broken down by time of system 
use, demonstrate that user satisfaction did not appreciably 
change with the increase in experience of users. Figure 5.15 
shows the level of satisfaction with CHCS verses time of 
system use. 
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Figure 5 . 15 



None of the work groups; administrative, physicians, 
or ancillary; exhibited any high correlation between time of 
system use and level of satisfaction of any factor by 
regression analysis. However, it is noteworthy to report the 
trends in the level of satisfaction for the work groups over 
time . 

The administrative work group respondents with less 
than six months system use report a satisfaction index of 
8.71. As the users gain experience with the system of six 
months, the level of satisfaction increases to 17.42. 
Satisfaction remains consistent, at 17.29, for users with 
greater than 11 months experience. Figure 5.16 shows the 
relationship of the satisfaction of administrative users to 
time of system use. 
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Administrative 

Satisfaction verses Time of Use 
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Figure 5.16 



On the other hand, physicians' level of satisfaction 
is consistent (satisfaction index 6.50-6.37) for users up to 
11 months of experience. The level of satisfaction increases 
for users with greater than 11 months of experience. Figure 
5.17 shows physician satisfaction verses time of system use. 
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Figure 5 . 17 



Unlike either the administrative or physician work 
groups, ancillary respondents with the least experience in 
system use demonstrated the greatest satisfaction 
(satisfaction index 16.90). The level of satisfaction 
declined to 11.44 for users with six to 11 months experience. 
Satisfaction further declined to 1.56 for users with greater 
than 11 months experience. Figure 5.18 shows the level of 
satisfaction verses time for ancillary users. 
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VI. ANALYSIS OF FINDINGS 



A. INTRODUCTION 

The purpose of this research is to describe and document 
the user information satisfaction characteristics of the users 
of the Composite Health Care System at the Naval Hospital, 
Charleston. The study results are not to be seen as a 
definitive evaluation of CHCS, but provide a baseline 
reference point for identifying system strengths or possible 
areas of conflict and dissatisfaction. Areas identified as 
dissatisfying should be examined further through interviews 
and by examining the development and operating procedures of 
the work group to determine the possible causes and course of 
action . 

The value of the satisfaction scores (positive or 
negative) is an important finding in itself, however, the 
relative comparison of scores across different groups of users 
is useful for the purposes of this study. 

The analysis of the survey findings will primarily focus 
on the differences and similarities among the three work 
groups: administrative, ancillary, and physicians . The first 
section will look at the overall user information satisfaction 
with CHCS. The second section will discuss significant 
differences in individual question responses reported between 
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work groups. The third section will compare the three work 
groups with respect to the grouped factors: local MIS staff 
and services, contractor services, knowledge and involvement, 
and information product. The last section will look at user 
satisfaction for the three work groups from the perspective of 
time of system use. 

B. OVERALL SATISFACTION 

1 . Combined Groups 

The overall satisfaction index is a general 
description of the user's perception of CHCS as a whole. A 
negative result index would indicate dissatisfaction while 
positive results index some degree of satisfaction with the 
system. The survey average satisfaction index reported is 
11.26. Since the overall satisfaction index ranges from -39 
to 39 in scale, the respondents average overall satisfaction 
is characterized as slightly satisfied using the adjective 
qualifiers of Baroudi and Orlikowski (1988) . 

2 . Comparison of Work Groups 

The three work groups used in this study were chosen 
to differentiate the users by the way they primarily use the 
computer system. There were no significant differences found 
in overall satisfaction between any one work group and the 
remaining groups. Although all groups are considered slightly 
satisfied with CHCS, there are meaningful differences between 
the work groups' overall satisfaction. 
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Lowest of the work groups in overall satisfaction was 
the physician group (8.07). Physicians use CHCS almost 
exclusively for the conduct of direct patient care. The 
functionality of the system for physicians is the most diverse 
of the three groups. Requirements of computer input are 
different for medical record queries, prescribing medications, 
laboratory tests, radiology requests, and so on. The 
physician needs to know the procedures for each type of 
functionality to use the system effectively. 

Each physician has a terminal located in the office. 
When using the system for out-patient purposes, the patient 
frequently is present in the office at the time of use. In 
addition, the medical profession have not traditionally used 
automated means to document patient treatments or write 
prescriptions in the past. The computer system can be 
considered, by them, an intrusion into their already busy 
schedule and has caused the physicians to alter their work 
patterns to accommodate the system. 

The significant change of work habits, extensive 
functionality requirements, time constraints imposed on the 
physicians, and the presence of patients in the office while 
operating the system combine to make this finding not 
surprising. 

Ancillary personnel, the largest of the work groups 
(46%) , demonstrated a slightly higher overall satisfaction 
score (9.65) . Most of the users in this work group are 
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technicians using the system to process prescriptions, 
laboratory specimens, or radiology requests. Ancillary areas 
in the hospital have had automated processing capabilities, in 
some form or another, for many years and are very accustomed 
to general computer use. Due to the large number of items 
processed each day in these areas, the users are under a tight 
time constraint to process items through the computer quickly. 

The administrative group scored the highest overall 
satisfaction scores. Administrative personnel use the system 
for purposes other than for the direct conduct of patient 
care. With a few exceptions, the work performed on the system 
can be accomplished without the pressure of a limited and 
specific time constraint. The computer system can therefore 
be used at a comparatively slower pace than the other work 
groups. Once again with a few exceptions, most of the work 
performed by the administrative personnel previous to CHCS was 
performed manually. Automating the compilation and generation 
of reports can significantly reduce the time spent on these 
activities . 

C. SIGNIFICANT INDIVIDUAL QUESTION DIFFERENCES 

Although there are demonstrated differences in 
satisfaction among the various groups in every question, only 
the questions with significant differences among the work 
groups will be discussed in this section. The individual 
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questions will be combined into the grouped factors 
representing satisfaction in the next section. 

1 . Degree of Training Provided to Users 

The results of the survey show the physicians were the 
least satisfied with the training provided, followed closely 
by the ancillary personnel. There is a significant increase 
in the satisfaction with user training expressed by the 
administrative personnel. 

Effective user training is extremely important to the 
success of any computer system. The person's satisfaction 
with the initial user training received sets the stage for his 
or her satisfaction while operating the system. Often, user 
training is intensive only during the time a system is being 
implemented. After implementation new users are frequently 
required to learn the functionality of a computer system with 
On-the-job-training (OJT) . Formalized training programs are 
difficult to conduct on a regular basis due to the constant 
change-over of personnel experienced at medical treatment 
facilities and the specialized requirements of the different 
work groups. Satisfaction with the degree of training 
provided is closely associated with the user's understanding 
of the system and their feeling of participation. 

2. User's Understanding of the System 

Users generally attain a high level of understanding 
of a computer system from either effective training or 
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experience or both. As anticipated, the work groups that 
reported a low level of satisfaction in user training 
(Physicians in particular) demonstrated a significantly lower 
level of satisfaction with their understanding of the system. 
The diverse functionality of the module used by physicians may 
also diminish their perception of the level of understanding. 

3. User's Feeling of Participation 

The user's perception of the shared involvement and 
commitment to the operation of the computer system is 
beneficial to the success of a computer system. Satisfaction 
with the level of participation naturally follows from the 
satisfaction of user training and the relationship with the 
MID staff. It is not surprising to observe the administrative 
work group scoring significantly higher than the other groups 
on this question. 

D . GROUPED FACTORS 

Baroudi and Orlikowski (1988) argue that the individual 
questions can be grouped into three factors, MIS staff and 
services, information product, and knowledge and involvement, 
that comprise user satisfaction. As discussed in Chapter V, 
the functions normally provided by a MIS of the companies 
studied in their research are provided by two separate 
entities at Naval Hospital, Charleston. Local support 
services are provided by the Command' s Management Information 
Department, whereas software support is furnished by a 
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civilian software contractor. It is logical to divide the 
single factor, MIS staff and services found in the original 
study, to make two separate factors: local MID staff and 
services and Contractor's services. 

The three work groups generally scored similarly in each 
of the four factors. The only exception being the 
statistically significant high level of satisfaction of the 
administrative group for the knowledge and involvement factor. 

The most notable finding when four factors are examined is 
the significant negative level of satisfaction of all the work 
groups with the services provided by the software contractor. 
The services specifically asked in the questionnaire dealt 
with satisfaction of the time for system changes and new 
system development. The negative satisfaction finding for 
this factor is, however, expected. Bailey and Pearson (1983) 
list these two factors as the two most frequent for causing 
dissatisfaction among users; a finding supported by other 
published reports. 

At the other end of the spectrum, the user's perception of 
the local MID staff and services is considered quite 
satisfactory. There does not appear to be any communication 
or attitudinal barriers to prevent an interchange of ideas 
between the respondents and the local MID staff. 

Lower in satisfaction is the user's perception of the 
information product factor. The information product measures 
the quality of the information output provided to the user. 
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The three work groups combined rated this factor as slightly 
satisfactory. The users perceive the system positively in 
regards to the information's accuracy, variance, reliability, 
and relevancy to the intended function. 

The physician and ancillary work groups displayed a 
positive satisfaction index for the knowledge and involvement 
factor. However, the satisfaction level demonstrated, albeit 
positive, was very close to neutrality. Administrative 
personnel indicated a satisfaction level significantly higher 
than the other two groups; high enough to be considered 
slightly satisfied with the factor. The combination of 
satisfaction with user training, system understanding, and 
participation is responsible for administrative work group's 
higher level of satisfaction.. 

E. SATISFACTION VERSES TIME OF SYSTEM USE 

Satisfaction with a computer system is expected to improve 
the longer users operate a system over time; gaining more 
experience and understanding of the system. 

An increase in overall satisfaction resulting from longer 
time of system use did not emerge from this study. The study 
did show, when considering the survey population as a whole, 
that user' s satisfaction remained constant when compared to 
length of time the respondent had used the system. 

As discussed in Chapter V, the Ancillary group making up 
46% of the survey population tends to influence all 
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generalizations drawn from total survey population findings. 
Breaking down the findings by work groups demonstrates this 
fact well. 

Positive trends in satisfaction for the administrative and 
physician work groups were displayed. Personnel in those work 
groups who have used the system the longest (greater than 11 
months) were more satisfied with CHCS than those who had used 
the system less than six months. 

The work performed by CHCS in the physician and 
administrative work groups was previously performed manually. 
As they worked with the system, their experience and 
confidence with CHCS expanded, thereby increased satisfaction 
was a result. 

Administrative personnel, satisfied with the training 
provided, quickly grasped the system procedures. The 
structured application of CHCS to their work performed 
assisted in their acceptance of the system. Rapid increases 
in satisfaction with CHCS resulted. 

The complex functionality of the physician modules is 
possibly responsible for their relatively prolonged time 
period before satisfaction is increased. After physicians 
learned the multitude of inputs necessary, their confidence 
and satisfaction with CHCS improved. 

Ancillary personnel findings indicate a steady decrease in 
satisfaction with CHCS for those who have used the system the 
longest when compared to those persons having less than six 
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months exposure to system use. This is the opposite to what 
was anticipated. It is not clear why this finding occurred. 
An in-depth analysis of their policies and procedure along 
with interviews with the personnel is necessary to determine 
the cause of this finding. 



60 



VII. CONCLUSIONS AND RECOMMENDATIONS 



A. MEASURING COMPUTER SYSTEM EFFECTIVENESS 

In an environment of scarce financial resources, 
organizations need to measure the effectiveness of computer 
systems to justify expenditures. Of the several direct and 
in-direct methods available to measure computer system 
effectiveness, measurement of user information satisfaction is 
a suitable, available and effective surrogate measure. The 
previously developed and validated user information 
satisfaction questionnaire of Baroudi and Orlikowski (1988) is 
easily administered and provides sufficient responses to 
determine a baseline reference point for overall satisfaction 
as well as indicating potential problem areas. The survey is 
not a definitive evaluation of CHCS, but combined with further 
investigation, can be a powerful tool in the analysis and 
interpretation into the cause of user dissatisfaction. 

B. SATISFACTION CHARACTERISTICS OF CHCS 

Overall satisfaction is a generalization of all the 
characteristics that effect the satisfaction of the user. 
Along with overall satisfaction, the study specifically looked 
at the four factors that make up satisfaction: local MID staff 
and services, contractor services, knowledge and involvement, 
and information product. 
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1. Overall Satisfaction 



The survey results indicate the users are slightly 
satisfied with CHCS at Naval Hospital, Charleston as a whole. 
The overall satisfaction scores do not differ significantly 
for the various types of personnel operating the system. In 
other words, the different ways (modules) in which people use 
the system did not affect their overall perception of the 
system. 

2 . Areas of Satisfaction 

All three work groups rated the local MID staff and 
services as quite satisfactory. This is an indication that 
the local MID personnel is interacting well with the system 
users. The users perceive the local MID is taking care of 
their needs. 

The output of the system, or information product, is 
considered by the users as slightly satisfactory overall. 
What they receive from the system in the form of output meets 
their expectations. 

Although the overall satisfaction scores did not 
differ significantly, the administrative work group displayed 
higher satisfaction in many areas. The slightly higher 
satisfaction can be traced to their higher perception of 
satisfaction with the training provided. This leads to higher 
satisfaction in participation and understanding of the system. 
The manner in which the administrative group personnel are 
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trained or assisted is producing a more positive effect on 
their satisfaction than the other work groups. 

3 . Areas of Dissatisfaction 

Factors that result in user dissatisfaction or 
significantly lower satisfaction when compared to other 
factors are potential areas for concern. Further 
investigation into areas of dissatisfaction can provide 
context, history and insight for possible corrective measures. 

The respondents were most dissastified with the 
contractor services in respect to the time for new system 
developement and changes to the system. Although the exact 
cause of this dissatisfaction cannot be determined, most 
likely, the user' s perception of the time required for these 
contractor activities is much less than the reality of the 
situation . 

4 . Satisfaction with Time of System Use 

The expectation is that users will become more 
comfortable and confident, and therefore more satisfied, with 
CHCS as they gain increased experince using the system. This 
expectation was shown true for the physician and the 
administrative groups. However, the ancillary work group's 
satisfaction with the system was lower for experienced users 
than the less experienced users. 
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C . RECOMMENDATIONS 



This study showed that the short-form questionnaire 
developed by Baroudi and Orlikowski (1988) is an effective 
means to measure user information satisfaction. The survey 
instrument is an appropriate instrument to document user 
satisfaction with the CHCS at other military medical 
facilities as well as documenting changes in user satisfaction 
at Naval Hospital, Charleston at a later time. 

A survey of this type is best conducted by an individual 
at the site. It is beneficial for the researcher to have 
first hand experience as a user of CHCS. Additionally, direct 
contact with individuals for interview by the researcher is 
beneficial for adding context to history. 

Additional investigation should be undertaken in several 
areas to add context and history. A larger and clearer 
picture may explain the reasons why the work groups had 
significant differences in their reported satisfaction to the 
various factors. These include: 

1. The administrative group's overall higher satisfaction 
over the other work groups 

2. The physicians' lowered perception of satisfaction in 
almost all areas 

3. The reasons for the respondents having negative 
perceptions of satisfaction with contractor's services 

4. The reduction of satisfaction for experienced 
ancillary users from less experienced ancillary users 
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The Baroudi and Orlikowski short-form questionnaire (1988) 
should be used to measure UIS at other CHCS sites for 
comparison to the results obtained from this study at Naval 
Hospital, Charleston. The survey should also be conducted at 
a later time at Naval Hospital, Charleston to monitor for any 
changes in UIS. 
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1 . 



APPENDIX A 

Part A: General Information 

Hospital Department: (Check one) 

General Administration 

Nursing Administration 

Dietary 

Emergency 

Laboratory 

Medical Clinic 

Inpatient Nursing 

Pharmacy 

Radiology 

Other (Specify) : 

2. Job Description: (Check one) 

Corpsman (0000) 

Technician 

Nurse 

Pharmacist 

Physician 

Physician Assistant 

Other (Specify) : 

3. Highest Level of Education: (Check one) 

High School Graduate 

Some College 

Bachelor's Degree 

Some Graduate Work 

Master' s Degree 

Doctoral Degree 

Medical Degree 

Other (Specify) : 

Age : 

Gender: Male Female 

Length of time (in months) you have used CHCS : 



Have you used other computer systems before ? Yes 

If your answer was Yes to question 7, was it a health care 
information system ? Yes No 



4 . 

5. 

6 . 

7. 

8 . 
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Part B: The Questionnaire 

This section of the survey conveys your own personal feelings 
concerning the use of the Composite Health Care system at Naval 
Hospital, Charleston. Please do not attempt to analyze the 
questions. Remember, there are no right or wrong answers. 

Please follow these instructions: 

1. Check each scale in the position that describes your evaluation 
of the factor being described. 

2. Check each scale, do not omit any. 

3. Check only one position for each scale. 

4. Check in the space, not between spaces. THIS, NOT THIS 

: X : X : 

5. Work rapidly. Rely on your first impressions. 

ANSWERS BASED ON YOUR OWN FEELINGS 

1 . Relationship with the Management Information Department (MID) staff 

dissonant : : : : : : : : harmonious 

bad : : : : : : : : good 

2 . Processing of requests for changes to existing systems 

fast : : : : : : : : slow 

untimely : : : : : : : : timely 

3 . Degree of training provided to users 

complete : : : : : : : : incomplete 

low : : : : : : : : high 

4. User's understanding of systems 

insufficient : : : : : : : : sufficient 

complete : : : : : : : : incomplete 

5. User's feeling of participation 

positive : : : : : : : : negative 

insufficient :::::::: sufficient 
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6. Attitude of the Management Information Department staff 

cooperative : : : : : : : : belligerent 

negative : : : : : : : : positive 

7 . Reliability of output information 

high : : : : : : : : low 

superior : : : : : : : : inferior 

8 . Relevancy of output information (to intended function) 

useful : : : : : : : : useless 

relevant : : : : : : : : irrelevant 

9 . Accuracy of output information 

inaccurate : : : : : : : : accurate 

low : : : : : : : : high 

10. Precision of output information 

low : : : : : : : : high 

definite : : : : : : : : uncertain 

11 . Communication with the Management Information Department staff 

dissonant : : : : : : : : harmonious 

destructive : : : : : : : : productive 

12 . Time required for new systems development 

unreasonable : : : : : : : : reasonable 

acceptable : : : : : : : : unacceptable 

13. Completeness of the output information 

sufficient : : : : : : : : insufficient 

adequate : : : : : : : : inadequate 

Thanks again for your cooperation 
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APPENDIX B 



Composite Health Care System 
User Satisfaction Survey 



This survey is part of a study of the Composite Health Care System 
(CHCS) effectiveness in military hospital inpatient, outpatient and 
administrative settings. The purpose of the study is to obtain information 
about your perceptions of how well CHCS functions in your area. 

Just a few minutes are required to fill out the brief general 
information and the 13 question survey. Your responses will be treated in 
complete confidentiality. The data gathered through the survey will 
provide valuable insights into the system's strengths and weaknesses, and 
assist in future development, training and use. 

The success of this survey depends on receiving as many completed 
surveys as possible from users in all hospital areas. Your participation 
makes an important contribution towards this end. There are no right of 
wrong answers, only your candid response to each question. Please do not 
emit any of the questions in either section of the survey. 

Please return your completed survey to LT Powell in the Pharmacy 
Department in the addressed envelope provided. Thank you for your 
cooperation. 



LT L.E. Hurd, MSC 

Naval Postgraduate School 

Monterey, Ca 



APPENDIX C-l 



COMBINED RESULTS FOR DOCTORS VS ADMINISTRATIVE VS ANCILLARY 



TOTAL SATISFACTION 
DOCTOR 
ADMIN 
ANCILLARY 



AVERAGE INDEX 
0.620879 8.07 
1.183258 14.96 
0.742475 9.65 



INDIVIDUAL QUESTION RESPONSES 



QUESTION 


1 


2 


3 


4 


5 


DOCTOR 


1.380952 


-0.45238 


0.047619 


0.142857 


0.52381 


ADMIN 


1.897059 


-0.08824 


1.426471 


1.411765 


1.588235 


ANCILARY 


1 . 652174 


-0.82609 


0.184783 


1.032609 


0.913043 


STAND DEV. 


DOCTOR 


1.28694 


1.328898 


1.342593 


1.601393 


1.611515 


ADMIN 


1.198752 


1.967916 


1.586474 


1.498609 


1.30601 


ANCILARY 


1.287597 


1.700735 


1.803107 


1.712598 


1.691827 



QUESTION 

DOCTOR 

ADMIN 

ANCILARY 


6 

1.571429 

2.308824 

1.934783 


7 

1.071429 

1.029412 

0.48913 


8 

0.571429 

1.220588 

0.902174 


9 

1.404762 

1.117647 

1.097826 


10 

0.857143 

0.808824 

0.206522 


STAND DEV. 

DOCTOR 

ADMIN 

ANCILARY 


1.085219 
0.965948 
1 . 014232 


0.921104 

1.392631 

1.72566 


1.399353 

1.401977 

1.437793 


1.083345 

1.766432 

1.576334 


0.92582 

1.650485 

1.56569 


QUESTION 

DOCTOR 

ADMIN 

ANCILARY 


11 

1.357143 

1.941176 

1.576087 


12 

-0.92857 

-0.07353 

0.01087 


13 

0.52381 

0.794118 

0.478261 






STAND DEV. 

DOCTOR 

ADMIN 

ANCILARY 


1.185716 

1.144518 

1.178877 


1.471566 

1.764629 

1.775874 


0.833391 

1.550723 

1.660652 
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APPENDIX C-2 



PHYSICIANS vs 


THE REST OF THE 


SURVEY GROUP 


GENERAL 


AVERAGE 


INDEX 


DOCTOR SAT 


0.620879 


8.07 


STDEV 


1.445 


8.37 


NON-DOC SAT 


0.930288 


12.09 


STDEV 


1.703 


12.44 



GROUPED FACTORS 


A 

(MID) 


A 

(CONT) 


B 


C 


DOCTOR GROUP 


1.44 


-0.27 


0.885714 


0.065476 


NON-DOC GROUP 


1.86 


-0.69 


0.761773 


0.65 


DOC STDEV 


1.17 


1.82 


1.09207 


1.505041 


NON-DOC STDEV 


1.16 


1.41 


1.604875 


1.856174 



QUESTION 


1 


2 


3 


4 


5 


DOCTOR SATIS 
NON-DOC SATIS 
DOC STDEV 
NON-DOC STDEV 


1.380952 

1.75625 

1.28694 

1.252654 


-0.45238 

-0.5125 

1.328898 

1.849656 


0.047619 

0.7125 

1.342593 

1.81672 


0.142857 

1.19375 

1.601393 

1.631054 


0.52381 
1.20625 
1 . 611515 
1.578539 


QUESTION 


6 


7 


8 


9 


10 


DOCTOR SATIS 
NON-DOC SATIS 
DOC STDEV 
NON-DOC STDEV 


1.571429 

2.09375 

1.085219 

1.008123 


1.071429 

0.71875 

0.921104 

1.61039 


0.571429 

1.0375 

1.399353 

1.426997 


1.404762 

1.10625 

1.083345 

1.654409 


0.857143 

0.4625 

0.92582 

1.624837 


QUESTION 


11 


12 


13 






DOCTOR SATIS 
NON-DOC SATIS 
DOC STDEV 
NON-DOC STDEV 


1.357143 

1.73125 

1.185716 

1.174798 


-0 . 92857 
-0.025 
1.471566 
1.766032 


0.52381 

0.6125 

0.833391 

1.617466 
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APPENDIX C-3 



PHYSICIANS vs THE REST OF THE STUDY GROUP 
CALCULATED T -VALUE 

QUESTION T -VALUE 

1 -1.196 

2 0.169 

3 -1.869 

4 -2.666 * 

5 -1.734 

6 -1.991 * 

7 1.313 

8 -1.353 

9 0.998 

10 1.459 

11 -1.289 

12 -2.399 * 

13 -0.241 

FACTOR 

A -1.467 

MID 

A 0.982 

CONTRACTOR 
B -0.334 

C 1.331 

OVERALL 1.397 



* SIGNIFICANT AT ALPHA =0.05 
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APPENDIX C-4 



ANCILLARY vs THE REST OF THE SURVEY GROUP 



GENERAL 

ANCIL SATIS 
NON-ANCIL SATIS 
ANCIL STDEV 
NON-ANCIL STDEV 



GROUPED FACTORS 



ANCIL GROUP 
NON-ANCI GROUP 
ANCIL STDEV 
NON-ANCI STDEV 



AVERAGE 


INDEX 


0.742475 


9.65 


0.967832 


12.6 


1.722284 


11.7 


1.595243 


11.8 



A 


A 


(MID) 


(CONT) 


1.72 


-0.41 


1.82 


-0.32 


1.17 


1.78 


1.18 


1.73 



B C 



0.634783 0.328804 
0.935614 0.697727 
1.621374 1.874259 
1.395514 1.725329 



QUESTION 


1 


2 


3 


4 


5 


ANCILLARY 
NON- ANCILLARY 
ANC ST DEV 
NON-ANC ST DEV 


1.652174 -0.82609 
1.7 -0.22727 
1.287597 1.700735 
1.252887 1.753942 


0.184783 

0.9 

1.803107 

1.636454 


1.032609 

0.927273 

1.712598 

1.65185 


0.913043 

1.181818 

1.691827 

1.527616 


QUESTION 


6 


7 


8 


9 


10 


ANCILLARY 
NON-ANC ILLARY 
ANC ST DEV 
NON-ANC ST DEV 


1.934783 0.48913 
2.027273 1.045455 
1.014232 1.72566 
1.070532 1.229502 


0.902174 

0.972727 

1.437793 

1.430077 


1.097826 

1.227273 

1.576334 

1.542422 


0.206522 

0.827273 

1.56569 

1.413299 


QUESTION 


11 


12 


13 






ANCILLARY 
NON-ANC ILLARY 
ANC ST DEV 
NON-ANC ST DEV 


1.576087 0.01087 
1.718182 -0.40909 
1.178877 1.775874 
1.189648 1.70426 


0.478261 

0.690909 

1.660652 

1.325442 
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APPENDIX C-5 



ANCILLARY vs THE REST OF THE STUDY GROUP 
CALCULATED T -VALUE 

QUESTION T -VALUE 



1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 


-0.18 
-1.653 
-1.98 * 
0.299 
-0.795 
-0.423 
-1.768 
-0.235 
-0.396 
-1.984 * 
-0.572 
1.151 
-0.674 


FACTOR 

A 

MID 


-0.406 


A 

CONTRACTOR 


-0.245 


B 

C 


-0.948 

-0.976 


OVERALL 


-1.197 



* SIGNIFICANT AT ALPHA = 0.05 
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APPENDIX C-6 



ADMINISTRATION vs THE REST OF THE SURVEY GROUP 



GENERAL 

ADMIN SAT 
NON-ADMIN SAT 
ADMIN STDEV 
NON- ADMIN STDEV 



GROUPED FACTORS 



ADMIN GROUP 
NON-ADMIN GROUP 
ADMIN ST DEV 
NON-ADMIN ST DEV 



AVERAGE 


INDEX 


1.182127 


14.96 


0.704363 


12.94 


1 . 6444 


9.16 


1.645581 


10.72 



A 

(MID) 


A 

(CONT) 


B 


C 


2.05 


-0.09 


0.994118 


1.084559 


1.63 


-0.8 


0.664463 


0.244403 


1.12 


1.86 


1.559052 


1.742729 


1.18 


1.68 


1.486632 


1.770465 



QUESTION 

ADMIN 1 
NON -ADMIN 1 
ADMIN STDEV 1 



897059 -0.08824 1.426471 1.411765 1.588235 
567164 -0.70896 0.141791 0.753731 0.791045 
198752 1.967916 1.586474 1.498609 1.30601 



NON-ADMIN STDEV1. 265863 1.751332 1.746904 1.676404 1.605592 



QUESTION 



10 



ADMIN 


2.308824 


1.029412 


1.220588 


1.117647 


0.808824 


NON-ADMIN 


1.820896 


0.671642 


0.798507 


1 .19403 


0.410448 


ADMIN STDEV 


0.965948 


1.392631 


1.401977 


1.766432 


1. 650485 


NON-ADMIN STDEV 1.04371 


1 .498369 


1.430456 


1.555407 


1.512961 


QUESTION 


11 


12 


13 






ADMIN 


1.941176 


-0.08824 


0.794118 






NON-ADMIN 


1.507463 


-0.28358 


0.492537 






ADMIN STDEV 


1.144518 


1.764629 


1.550723 






NON-ADMIN STDEV1. 183936 


1.74471 


1.487448 
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APPENDIX C-7 



ADMINISTRATION vs THE REST OF THE STUDY GROUP 
CALCULATED T -VALUE 

QUESTION T -VALUE 



1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 


1.259 
1.614 
3.599 * 
1.929 
2.503 * 
2.275 * 
1.161 
1.411 
-0.221 
1.213 
1.759 
0.53 
0.949 


FACTOR 

A 

MID 


1.719 


A 

CONTRACTOR 


1.935 


B 

C 


1.036 
2.265 * 


OVERALL 


0.938 



* SIGNIFICANT AT ALPHA = 0.05 



76 



APPENDIX C-8 



SATISFACTION BASED ON TIME ON THE SYSTEM 



OVERALL 
# PEOPLE 



TIME OF 

USE AVG SAT 



28 


1 


TO 5 


11.64286 


36 


6 


TO 11 


11.84722 


37 


>= 


» 12 


10.37838 


101 









ADMINISTRATION 

TIME OF 

# PEOPLE USE AVG SAT 



8 

9 

17 

34 



1 TO 5 8.8125 
6 TO 11 17.4375 
>= 12 17.36111 



ANCILLARY 

TIME OF 



# 


PEOPLE 


USE 


AVG SAT 




12 


1 TO 5 


16.91667 




19 


6 TO 11 


11.5 


15 

46 

DOCTORS 


>= 12 
TIME OF 


1.533333 


# 


PEOPLE 


USE 


AVG SAT 




8 


1 TO 5 


6.5625 




8 


6 TO 11 


6.375 




5 


>=12 


13.2 



21 
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